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A taxonony for literature reviews in education and 
psychology is presented. The increased use of the descriptor 
'"literature review" in ERIC and Psychological Abstracts documents 
between 1969 and 1983 is cited as creatind the need for 
categorization. The taxonomy c^egorizes reviews according to focus, 
goal, perspective, coverageyr^;$^anization , and audience. The seven 
winners of the American Educational Research Association's Research 
Review Award are used to illustrate these categories. Data on 
intercoder reliability of taxonomy codings when applied by readers is 
presented. The taxonomy is used to describe a representative sample 
of existent reviews. Suggestions are made concerning' kow the ^axonomy 
might facilitate judgments concerning the quality of future knowledge 
synthesis activities. General standards for evaluating reviews are 
presented. (DNH) 
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" • Abstract 

it taxonomy of literature reviews. -In education and psychology Is 
presented. The taxonomy categorizes reviews according to: . (a) focus; 
(b) goal; (cj perspective; (d) coverage; (e) organization; and 
(f) audience. The seven winners of the Americafi Educational Research 

Association's Research Review Award are used to illustrate the taxonomy's 

■ < ; 

categories. Data on th6 reliability of taxonomy codings when applied by 
- readers is presented* Results of a survey of >ev1ew authors provides 
baseline data on how frequently different types of reviews api^ar in the 
ediyratlon and psychology literature. How the taxonony might help in 
judging the quality of literature reviews Is discussed, along with more 
general standards for evaluating reviews. 
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The twa major abstracting services 'in psychology and education, the 
Educational Resources Information Center (ERIC) and Psychological 
Abstracts , use the term "literature review**, or a, variation thereof, to 
describe the documents they contain. Figure 1 presents the percentage of 
all documents in each database that were assigned the term "literature 
review" between the years 1969 and 1983. The trend for both databases is 
for an increasing percentage o^^ocuments to be assigned this descriptor 
over the period covered, with an exceptional expansion in the use of the 
term by Psychological Abstracts between 1979 and 1983. 



Place Figure 1 about- here 



The probable cause for the growing prominence of reviews in the 
education and psychology literature is the increased numbers of personnel 
and the accompanying information explosion that has occurred in these 
disciplines. According to Garvey and Griffith (1971): . .the 
individual scientist is . . . overloaded with scientific information and/ 
[can] no longer keep up with and assimilate all the information being 
produced that [is] related to his primary specialty" (p. 350). The 
response to this overload appears to be, first, a narrwing of 
specializations in which social scientists attempt to keep up with primary 
research and scholarship, and second, a greater reliance on literature 
reviews to remain abreast of developments in other fields o^ interest. 
Also, regardless of the assimilating capacities of social scientists, 
expanding literatures necessitate the collecting, evaluating, and ^ 
synthesizing of scholarship in order to bring coherence and perspective to 
problem areas. 
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The enhanced role of the literature review requires that- this 
expository form be given careful Scrutiny. To date, such exaininations have 
been scarce. The only aspect of literature Reviewing that has received 
prolonged attention is the integration of empirical research. This concern 
is primarily an outgrowth of the introductiorr of meta-analysis procedures 
(Glass, McGaw & Smith, 1981; Rosenthal, 1984). However, a survey'of recent 
authors of literature reviews, to be discussed momentarily, reveals that 
less than one-fifth of all reviews are undertaken for the purpose of 
exhaustively synthesizing a research literature. The majority of reviews 
are conducted for, other purposes, and these synthesis activities have been. 
almost completely neglected. 

In this paper, I will attempt to correct this omission by offering a 
general definition of the^ term "literature review" and a^^j^xbnomy for 
classifying literature reviews according to their major characteristics. I 
Will then illustrate the taxonomy by applying it to the past winners of 
AERA's Research Review Award. Some data on Intercoder reliability will be 
presented and the taxonomy will be used to describe j jj y'esentatlve/ sample 
of existent reviews. Finally, I will make some suggestions concerning how 
the taxonomy might facilitate judgments concerning the quality of future 
knowledge synthesis activities. 

Because the existing literature on literature reviews hardly forms the 
basis for a review itself, I have supplemented the prior works on this 
topic in two ways. First, I conducted in-depth, unstructured interviews 
with fourteen scholars in diverse fields of education and psychology who 
were conducting literature reviews. The interviews occurred at several 
points during the reviewing process and touched on all aspects of the task, 
from problem formulation to editorial remarks. Second, based on the 
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interviews and on input from numerous people involved in the generation of 
knowledge syntheses, including the directors of ERIC clearinghouses and the 
Nat1ona.l Institute of Educations' Dissemination in Practice program staff, 
a structured questionnaire was developed and completed by 68 scholars who 
had recently published reviews of research literatures (see Cooper, 1984a). 
Several of the results of this survey aided in the formulations I will 
present. 

Of course, this paper is not intended to be a definitive statement on 
the nature of literature reviews. Instead, it is to be a working document 
meant to stimulate future discussion of the goals, processes, and problems 
associated with the literature review and, by implication, to help 
practicing literature reviewers produce documents of maximum utility for * 
their audiences, 

A Definition pf the Literature Review 
I decided to begin the search for ^ definition of the term literature 
review by examining the definitions used by ERIC and Psych Jfeg leal 
Abstracts . In the The^aQrus of ERIC Descriptors , the descriptor term 
"literature review" was accompanied by the scope note "surveys" of the 
materials published on a topic" (ERIC, 1982, p. 143). The ERIC Processing 
Manual (Section 5: Cataloging: ERIC, 1982) contained the following 
definition for the literature review as a document type: "Information 
analysis and synthesis, focusing on findings and not simply bibliographic 
«l^|^ions. Summarizing the substance of the literature and drawing 
conclusions from it" (p. 85). 

The Thesaurus of Psychological Index Terms (American Psychological 
Association, 1982) provides no definitions for the document types it 
assigns in cataloging the literature. An Inquiry to the offices df 
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Psychological Abstracts revealed that the document term "literature review" 
had no specific or fonnal definition. Instead*, the definition of the term 
and its appropriateness for a document was left to the intuitive judgment 
of the indexer, with the proviso that the document had to be exclusively or 
primarily a literature review (that is, not also contain a report of 
primary data) for the term to be employed (D. Langenberg, personal 
conitiunication, March 14, 1984). As one lexicographer at ERIC put it: 
"You knows one when you sees one." The scope note for the descriptor 
"literature review" in APA's Thesaurus defined these docwnents as "surveys 
of previously published material" (APA, 1982, p. 96) and also stipulated 
that the document should be entirely or primarily a .literature review for 
the descriptor to be applied. 

Another potential source of definitions was journals that specialize 
in publishing literature reviews. To this end, the policy statements of 
the Review of Educational Research and the* Psychological Bulletin were 
examined. The Review of Educational Research policy statement says that 
the journal "contains integrative reviews and interpretations and 
educational research literatures on both substantive and methodological 
issues." Psychological Bulletin 's policy states the journal publishes 
"evaluative and integrative reviews and interpretations of substantive 
and methodological issues in scientific psychology.** Further, "Integrative 
reviews that suqinarize a literature may set fortJh major develo^ents within 
a particular research area, or provide a bridge between related specialized 
fields . . .". Finally, original theoretical statements that contain 
literature reviews are not considered the province of Psychological 
Bulletin , but literature reviews that "develop an integrative theoretical 
s tatement^-^lfe^^ceptable . 

/ 
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It seems clear that a general definitib^ of a -literature review nfUst 
contain at least two elements; First, a literature" review uses as its 
database reports of primary or original scholarshfp, and does not report 
new primary scholarship itself. The primary refpdrfs used in a literature 
review may be verbal, but in the vast majority of cases reports are written 
documents. The types of scholarship may be empirica-1, theoretical, 
critical/analytic, or methodological in nature. 

Second, a literature review seeks to describe, sumnarize, evaluate, 
clarify, and/or Integrate the content of the primary reports. This second 
part of the definition implies that literature reviews are generally 
inductive in nature, a quality made explicit in the Psychological 
Bulletin 's definition. However, the relation between existing theories and 
literature reviews is not that simple. For instance, sometimes the 
documents being evaluated and integrated in a literature review are 
themselves theoretical statements or other literature reviews. Other 
times, theoretical positions form the framework for evaluation and 
integration, thus rendering the review more hypothetl co-deductive in 
character. This issue leads away frcnn the problen of how generally to 
define the form to the problem of how to distinguish among different types 
of literature reviews. 

Types of Literature Review 
Previous attempts at defining types of literature review have 
primarily been concerned with the foci and goals of reviews, with 
particular attention paid to reviews that sumnarize empirical research. 
For instance, Jackson (1980) offered the following set of goals: 
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Some (reviewers) are primarily interested in sizing up new 
substantive and/or methodological developments in a given field. Some 
are primarily Interested in verifying' existing, theories or developing 
new ones, Some are interested in synthesizing knowledge from 
. different lines of research, and still others are^imarlly interested 
in inferring generalizations about substantive issues from a set of 
studies directly bearing on those Issues (p. 438). 
To this list might be added Taveggia's (1974) notion that reviews are ' 
meant to highlight important issues that research has left unresolved, ^d ' 
Price's (1965) notion that reviews are meant to replace papers that have 
fallen behind the research front. 

These definitions provide a starting point for a nwre exhaustive 
taxonomy. They highlight some of the central foci and goals of reviews. 
Because they deal primarily with Integrative research reviews, however, 
they do nat capture many varying aspects of the documents that fall within 
the definition of literature review provided earlier. Therefore, I would 
like to systema,tl2e and expand on these foci and goals and also suggest 
several other characteristics that usefully distinguish annng literature 
reviews. These Include: the perspective of the reviewer; the intended 
coverage of the review; the organization of the review; and the review's 
intended audience. Table 1 presents the six characteristics and their 
related categories. I will describe each briefly, then demonstrate how the 
taxonomy can be applied. 
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' Focus. The focu5i of a review concerns the material that Is of central 

s 

\ 

interest to the revieweK^ Most reviews in e?iucation and psychology center . 
on one or more of four arenas: research outcomes, research methods, 
theories, and practices or appTications. The four feci are self 
explanatory and familtar to sois^al scientists. Of course, they are not 
mutually exclusive areas of interest; in fact, it is rare for a review to 
have only a single focus. Instead\ most reviews will have two or three 
foci that are given* varying degrees of attention. 

Goals . The second characteristic of a review is its goals. Goals 
concern what the author hopes the review will accomplish. The most obvious 
goal for a review is to integrate or synthesize pas.t literature that is 
believed to relate to the same Issue, In fact, this goal is so pervasive 
among reviews that it Is difficult to find reviews that don't attempt to 
synthesize works at some level. 

In their article on types of synthesis. Strike and Posner (1983) 
identified numerous activities that could be counted as integrative and 
that are often performed by literature reviewers. These Include 
(a) formulating general statements from fliultlple specific instances, a type 
of synthesis common in research reviews, (b) resolving the conflict between / 
contradictory ideas or statements of fact by proposing a new conception 
that accounts for the inconsistency, and (c) bridging the gap between 
theoriesvor disciplines by creating a common linguistic framework. 

While synthesis is pervasive among literature reviews, reviews can 
have other goals. For instance, reviewers may write for the purpose of 
critically analyzing the existing literature. Many reviews are judgmental 
about the work they focus on, be it research, theory, or practice. The 
intention of thes6 reviews is usually to demonstrate that past conclusions 
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.derived from the literature were unwarranted. The conclusion of 
unworthiness is typically based on the literature's incommensurabili ty w-ith 

f 

the reviewers' theoretical stance and/or criteria for methodological 
validity. Unlike a synthesis, a review that concentrates on criticism less 
often compares the covered literature one to another' but instead holds each 
instance up against a criterion and finds it either acceptable or not. 

A third goal that is often at the heart of reviews is to identify 
issues central to a field. These issues may involve (a) questions that 
have dominated past endeavors, (b) questions that should dominate future 

. ' * * » 

endeavors, or (c) methodological problems that have prevented a topic area 
from progressing. While reviews emphasizing central issues usually provide 
suggestions about how problems and controversies in an area might be 
overcome, they are not necessarily syntheses because they don't always 
formulate generalities, attempt tp resolve conflict, or suggest bridges 
between areas. However, as with foci, reviews more often than not have 
.multiple goals. Frequently, integration and criticism or integration and 
identification of central issues go hand in hand. 

Perspective . A third characteristic that distinguishes reviews 
concerns the point of view the reviewer employs in discussing the 
literature. Two perspectives can be identified: (a) neutral or 
dispassionate representation and (b) espousal or advocacy of a position or 
perspective. In the former, the reviewer tries to distill the works in a 
topic area with as little personal interpretation and evaluation as 
possible. The attention given to different theories, methods, issues, or 
outcomes ii meant to reflect their relative prominence in the pertinenV^ 
literature. In essence, the revie*«r attempts to play the role of an 
"honest broker." * 
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The second perspective is tlie-opj)Osite of the first. Here, the 
reviewer deliberately plays the role of an advocate and espouses the 
virtues of a particular paradigm^ theory, foethodology , or practice. The 

reviewer undertakes the task of accumulating and synthesizing the 

♦ ■ 

literature^n the service of demonstrating the value t)f a particular point 
.of view. 

While Table 1 presents the two perspectives as separate approaches, it 
is probably best to think of them as opposite ends of a continuum. We 
might label the continuum "degree of active construction" by the reviewer. 
•Whether reviewers can in fact be "honest" brokers" is a debate receiving 
considerable attention among philosophers of science, and the argtflnents 
need not be reiterated here (see Philips, 1983; Eisner, 1983). However, it 
is important to note that while reviewers cannot maintain both t|ie 
descriptive and interpretive perspectives simultaneously, they can switch 
modes within the same work, first describing dispassionately the contents 
of an area and then applying a particular perspective to it. We shall see, 
however, that such approaches are rare— most authors opt for one end of the 
continuum or the other and maintain that stance throughout the review. 

Coverage . The next characteristic, coverage, is probably the most 
distinct aspect of literature reviewing. The extent to which reviewers 
find and include relevant works In their papfer is the single activity that 
sets this expository form apart form all others. How reviewers search the 
literature and how they make decisions about the suitability and quality of 
material involve methods and analytic processes that are unique tf this 
form of'jH^Khip (see Cooper, 1984a). 
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The^lPVogy distinguishes between four types of coverage. The first 

level, exhaustive coverage, means thcv-reviewer intends to be cmnprehensive 
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in the presentation of works relevant to the topic under consideration. An 

V 

effort is made to include the entire literature or most of it, not just a 
sanple, and to, base" conclusions and discussion^ on this aVl-inclusive 
information base. In this type of papeoi' the autbor describes' all 'the 
works relevant to the conclusions thai^are drawn, but perhaps not in grpat 
detail. • 

- .The second type of coverage a] fo bases conclusions on entire 
literatures but only a selected sample of works are actually described in 
the paper. The strategy for selecting works to cite might follow either of 
the patterns to be described momentarily.^ Especially in research 
Integrations, authors often formulate conclusions in very general terms, 
using, for^Hpnce, pKrases like "In sunmary, the research indicates 
. . or "The literature on this topic reveals . . Such statements 
imply a compreh^psive cqverage, but not necessarily that the work cited in 
the text exhausts the Titef»ture. 

^ .^i^ tfie reader's perspective the distinction between exhaustive ; 
coverage and exhaustive coverage with selective citation is important. A ^ 
reviewer who presents the entire informatiorf base allows the reader to 
evaluate (a) whether the coverage was, in fact, exhaustive and (b) whether 
the conclusions are warranted by the works Included. .The reviewer who has 
drawn general conclusions but only cites selected works (or mak^s no claim 
concerning how cited material was chosen) does not allowtthe reader to 

> 

perform such an evaluation. 

Some rev1e*i«rs will opt for a third Coverage strategy— presenting 
works that are representative of m^ny other works in a field. A sample Is 
presented that typif'ies larger groups of material. The author discusses • . 
the characteristics that make the sample illustrative of the larger group. 
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In this strategy, the author- freely choses the particular works that 
are deemed representative b^t the classes of material that need to be 
attended to are really not witShin the reviewer's discretion. Instead, 
this is a function of the frequency with which works that share "particular 
characteristics appear in the. literature. ' 

In the final coverage strategy, the' reviewer concentrates on works 
that have been central or pivoifal to a topic area.. This nay include 
materials , that initiated a line of investigation or thinking, that 
changed how questions were framed,, that introduce^ new methods, that 

• * 

engiendered important debate, pr that >perf ormed a heuristfc function 
for other scholars. Rather than being representative, a review .th^t 
covers pivotal works describes important initial efforts that have 
provided direction for a field. ' . • 

As, with the previous characteristics, a particular review can 

♦ * 

eiRploy more than drie coverage strategy. Obviously, the exhaustive 
and exhaustive/selective stratfgies are mutually exclusive, at least 
within the same topic doR&in. HoV^ver, it may not be uncommon for 
the representative and pivotal strategies to occur together. 

Organization . How a paper is organizjed is. a fifth characteristic 
that differentiates research reyiews. Reviews can be arranged 
(a) historically, so '^that topics. are Introduced in the chronological 
order in which they appeared in the literature, (b) conceptually, so 
that works relating to the same abstract ideas a|>pear toget^r^ or 
(c) methodologically, so that works employing similar methods are 
grouped as subtopics. Reviews can cofik>ine organizations by, for 
example, addressing works historically within a given conceptual or 
methodological franework. . 
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Audience . Finally, tKe intended audiences of reviews can differ from 
one another. Reviews can be written for groups of specialized researchers, 
•genteral researchers, practitioners, policy makers, or the general public. , 
The aud^^nce distinction probably manifests Itself most cljearly through the 
writing style of the reviewer. As reviewers move from addressing 
specialized researchers to addressing th^ general public, they employ Tess 
jargon and detail while often paying greater attention to the implications 
of the work being covered. Of course, it is rare to find literature 
reviews that speak directly to the general public. Instead, reviews 
written for more specialized audiences are sometiii^s distilled and 
simplified by popular writers before appearing tn' periodicals Intended for 
large general audiences. ^ . ' 

' * ^ • . Applying the Taxono^ to Award-Winning Reviews 

In order to illustrate how the taxonomy can be applied, and to uncover 
problems in its application, my two research assistants, Oavid Tom and Ron 
Ribble, and I undertook the pleasant task of reading the seven reviews that 
have won AERA's Research Review Award. We independently attempted to . 
describe each of the reviews by using the characteristics and categories in 
Table 1. Table 2 presents the fruit of our labor. Contair^ed in each cell 
are those categories that at least two of us agreed pertained to the 
review. Half of ttie listed categories received three votes and half 
received two votes. One of every eight category nwninations received only 
one vote. These are not listed. Before examining the table. It will be 
instructive to detail how the taxonomy was applied to one of the reviews.. 
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Noreen Webb (1982), the 1984 winner -of the Research Review Award, 
performed a review concerning student interaction in small learning groups. 
Mebb's focus was to "examine research bearing on the relationship between 
Interaction and achievement and research exploring the predictors of 
Interaction in small groups" (p. 422). At the end of the paper, some, but 
considerably less, attention was given to research methods through Webb's 
discussion of interpretive problems arising from "noncomparable designs, 
lack of detailed or appropriate observation procedures, inappropriate unit 
of observation and simplistic analytic strategy" (p. 439). 

The Integration goal of Webb's review Is exemplified by her use of 
sumRa ry statements such as "the research relating interaction in groups and 
.achievement generally shows that giving help and receiving help are 
positively related to achievement, and off-task and passive behavior are 
n^atlvely related to achlevenwnt" (p. 427). At the end of her paper, she 
Identifies central Issues in the area, based primarily, on her assessment of 
ri&search design and obs^vational problems with past research. 

Though she clearly believes Interaction variables can enhance our 
understanding of small group learning, Webb takes a generally neutral 
ppsitien toward the research meant to demonstrate this contention. In one 
Instance she calls the research "not sufficiently consistent at this time 
to warrant an unqualified conclusion" (p. 441). 

One problem that arises in applying the taxonomy ^s 111u%rated by our 
approach to the first three categories. Taxonony users are faced with the 
decision of whether to apply the categories from the perspective of a 
reader or from the inferred perspective of the author. In sdset instances 
the category nominations might differ. Thus, a reviewer might claim 
neutrality toward an area but a reader might perceive the paper is 
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an example of advocacy. In our application, we attempted to infer the 
intent of the author when making our jud^nents. Either approach can be 
employed, however, and ^n interesting set of issues arise concerning 
disparities between an author's expressed intentions and what they 

r 

accomplish. These issued relate to judgments of review quality, to \$i 
discussed later. ♦ 

kHth regard to coverage, Webb attempted to be fairly exhaustive, 
within the limiting criteria of only including studies Involving 
Individual learning and systematically measured Interaction (pp. 422- 
423). Applying the coverage categories Ted us to consider whether a 
literature revl^ can ever be tr^uly exhaustive. All authors of revlew^^ 
must necessarily exclude a multitude of work that lies near the boundary 
of their problem domain, works that other reviewers might choose to 
Include. To solve this problem in applying the latel, we chose to 
operationally define "exhaustive** as meaning comprehensive coverage 
within the limitations of the author's definition of the area. We 
also chose to label as exhaustive reviews that confined themselves to 
particular tlnre periods, for example all research conducted after 1575, 
if the author conprehensively examined the delineated period. Other 
users of the taxonomy might choose to operationally define exhaustiveness 
in a different manner. . 

Webb's review was organized by grouping studies that shared the 
same conceptual underpinning, though the concepts might be termed narrowly 
abstract. For example, her categorization of research under headings such 
as "helping behavior", "off-task and passive behavior", "ability groups 
composition and reward structure" are concepts closely tied to observable 

r 

measurement procedures. In discerning an author's organization schen^, we 
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found a good indicator was the headings employed to distinguish subtopics 
within the paper. 

Finally, the level of specificity of study descriptions and the fact 
that Webb's discussion focused on how small group research might best be 
conducted In the future indicated that specialized researchers were clearly 
her pirimary audience. 

Persons examining Table 2 to discover the kiey to writing an 
award-winning review will probably be diisappointed. Besides an emphasis on 
research integration and Identification of central Issues, the foci and 
goals that generally define the conH)et1t1on, there is little consistency 
across the seVen papers. In fact, even within the focus and goal 
categdrles the pipers are not homogeneous*' Shavel son, Hubner and Stanton's 
(1978) paper was primarily a methodological critique of self-concept 
measures and Thomas' (1982) paper was clearly me^nt to take Issue with the 

m 

back-to-basics movement. Boruch and Wortman (1979) focused on - 
methodological 'Issues 1n*evaluat1on research. 

The award committees have shown no preference for either neutral or 
advocacy-type papers, nor for a particular coverage strategy. The 
organization of papers has primarily been conceptual and the audience 
primarily specialized researchers. These consistencies, however, are 
reflective of how often these characteristics appear within the domain of 
all reviews, as we shallsee shortly. 

In sum. It appears that reviews of diverse form can be judged to be of 
the highest quality. The point Is Important because it underscores the 
nonjudgmental character of the taxonomy. In fact, the' omission of quality 
criteria from the taxonomy Is deliberate. Not to beg this Important 
question, however, I will later return to the problem of what makes a 
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quality review and how the taxonomy can facilitate quality judgments. 
First, however/ 1 would like to describe two more applications of ttie 
taxonomy. One concerns how reliably the scheme can be applied to 
describing reviews and the other concerns how often different types of 
reviews actually appear within the fields of education and psychology. 
Assessing the Reliability of Category Placements 
While the main purpose of the taxonomy is to catalog the various 
features of present-day reviews, the scheme would be of added utility If 
the category labels could be applied to reviews in a reliable manner, that 
Is, with a high degree of consistency across readers. To test whether th^is 
was the case, I asked my research assistants, Tom and Ribble, to read and 
categorize 37 literature reviews In psychology and education. The reviews 
were chosen from computer printouts of all documents published during the 
first six months of 1983 that were given the descriptor "literature review" 
by ERIC or Psychological Abstracts . 
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Table 3 presents the Intercoder reliability, 8»asured by Cohen's 
Kappa, and the percent-agreement for each of the six review 
characteristics. The "First Code" columns relate to the reader's choice of 
the primary category placement for each characteristic. The "First Plus 
Second Code" colusms define agreement as occurring when toth readers 
nominated a category as either the primary or secondary characteristic of 
the review. As an example, asswue one reader ^ald a review's primary goal 
was integration and secondary goal was criticism whil^the other reader 
said criticism was primary and lotfisratlon secondary. For the "First Code" 
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analysis this would be considered a disagreement. For the "First Plus 
Second Code** analysis this wOuld be consideretf two agreements. 

the results. are not very Encouraging. For first codes, the ^Kappa's 
ar^ unacceptably low. In the case two categories, goal and coverage, 
the low reltabllitles signify the readers were .able to agred on only about 
half of their judgments. For pio other categories .-organization and 
audience, the significantly lower values for Kappa than for 

r 

percent-agreement Indicate that. most codes fell Into only one category, a 

conceptual orgaplzatlpn and a specialized researcher audience. 'Kappa, In 

these Instances, adjusts downward the percent-agreement rates to account 

for what could have been concordance bas^d singly on reflated use of these 

single categories rather than any "tru^" discrimination by the readers (see 

*' ■ * ■ . 

Flick and SImmeU 1978). 

The Kappa's for first plus second codes are somewhat more Inspiring, 
based partly on a more even distribution of codes across categories and 
partly because the definition of agreement was less strict. This latter 
irifluenc^ is especially pronounced on the results for the goal category. 
That is, the two readers categorized many reviews as having the twin goals, 
of Integration and Identification of central Issues. They had difficulty, 
however, agreeing on which goal was primary and which was secondary. 

One might conclude frcnn the reliability data that the taxonomy is 
poorly defined or does not capture significant distinctions among reviews. 
I think there are good arguments against such a conclusion. First, the 
categories are the inductive product of interactions with scholars actively 
engaged in the reviewing process. Second, few of over one hundred 
reviewers who were asked to describe their own reviews using the taxonwny 
have objected to the categories or suggested different ones. 
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Instead of a lack of clarity in the definitions, I think the low 
reliabilities reflect the fact that most reviews require multiple codings 
for several of the categories. For instance, most reviews have more than 
one focus and goal. Thus, coders are often asked to make subtle 
distinctions in emphasis. Also reflected in Ihe low reliabilities is a 
lack of declarative statements on the part of the reviewers concerning what 
their papers are meant to accomplish and how they were constructed. This 
lack of information is especially dramatic in the case of thft revifewer's 
coverage strategy, the category which gave the readers the greatest 
difficulty. To illustrate, Jackson (1980) reported that of 36 reviews 
randomly chosen from prestigious social science journals, only one gave any 

* 

indication of the indexies and information retrieval systems used to search 
the literature and only seven indicated whether they Tiad analyzed the full 
set of studies or a subset. Such information would clearly help readers 
discern the intended coverage of a review. 

Regardless of the sources of the disagreements, the low Reliabilities 
indicate that if the taxonooiy is to be applied by readers, the consensus of 
multiple readers will be necessary to actomplish a trustworthy 
categorization of reviews. 

A Survey of Literature Review Authors 

While readers may have difficulty categorizing reviews, the authors of 
reviews should find the taxonomy adequately describes their intents and 
practices. To discover if this was the case and also to obtain some 
baseline data on how frequently different types of reviews appear in the 
education and psychology literature, I undertook a survey of recent review 
authors. The sample for the survey was generated by conducting a conpu ter 
search of ERIC and Psych Info in which all documents assigned the descriptor 
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"literature review" were retrieved that had been publish^ during 1984 and 
were on-line by December 4, 1984. For ^RIC, 168 such jdocwne'nts existed 
with publication dates through June 1984. For Psych Info . lOO documents 
through May 1984 were found. I then excluded 43 documents from ERIC and 
four from Psych Info because their abstracts indicated they v<e^ primarily 
annotated bibliographies ot project reports. Of the remaining 125 ERIC 
documents, my research assistants retrieved the addresses of 65 randomly 

chosen first authors from our university library.. Of the 96 Psych Info 

"i 

documents, 75 first authors were sent questionnaires. 

The questionnaire described the taxonomy to authors and asked them to 
rank order, within each characteristic, those categories that applied to 
their review while leaving blank those categories that were Irrelevant. Of 
the 140 questionnaires that were mailed, 108, or 77%, were returned 

completed, seven were returned undelivered, and five were rAurfied with an 

.••_■•> 

author comnent that their paper was in fiict not a review. 

The categories provided to authors appeared to adequately capture the 
vast majority of review characteristics. Authors infrequently made use of 
the opportunity to provide their own categories to describe their reviews. 
For example, 12 authors supplied their own description of focus and four of 
these were simply more specific depictions oiF categories provided in the 
taxonony. No author-offered focus was repeated more than once. Perhaps 
most troubling to me was the one author who described his wrk as a 
"nonempirical , nontheoretical assessment of reality." This response left 
me wondering why I was studying literature reviews rather than writing such 
papers myself! 

The most curious finding regarding author comments concerned the 
twelve authors who supplied self-definitions of perspective. I found that 
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most of these could be reclassified as descriptions of foci or goals. 

* ■ ♦ 

While 'Percentage of these misunderstandings was small and most of the 
authors providing self -described perspectives also ranked either the 
neutral or espousal al ternatives,. it would be informative to know if the 
misuse of this category was caused by an unclear definition or by a 
reluctance on the part of the authors to assert their perspective. I 
suspe'^t the latter was more often the case. 

None of the other characteristics lead more than six. percent of 
reviewers to supply descriptors that were other than specif ications'of 
categories already in the tdxonomy and no consistency in author-jjffered 
descriptors was evident.. This indicates that the addition of more 
categories to the taxonomy is probably unnecessary. In- general then, the 
responses of authors were more encouraging then the reliability of reader 
codings. Authors apparently felt comfortable describing their-reviews in 
the taxonomy's terms. 

Perhaps the best testimony to the taxonomies robustness came from a 
group of ten reviews abstracted by ERIC that had appeared In a Journal 
called Analytic Chemistry . I debated for some time over whether these 
papers should be Included in the san^le, given their somewhat exotic 
topics, such as "dynamic electrochemistry" and "atomic absorption, atomic 
fluorescence and flame emission spectometry." I decided to include the 
papers and only one author ret^f^ned the survey saying his reply would be 
inappropriate. All nine other chemistry authors returned the completed 
questionnaire without c(»nment. 

Table 4 preseRts the descriptive results of the survey. The first two 
columns list the characteristics and categories. Columns three and four 
list the percent of respondents who chose each category as a primary or 

23 



Literature Reviews 23 

secondary description of their review. The final column presents the 
number of reviewers who omitted the characteristic entirely from the 
description of their paper. 

Place Table 4 about here 



The responses revealed that about half, of all reviews priraaVily 
focused 'oh research outcomes, and three of four paid some attention to 
empirical results. One in five primarily focused on practical 
applications, and an equal nunber focused on theory. Only one review in 
ten took as its primary focus attention to research methods. 

The most frequent goal of a review was to critically analyze the 
relevant literature, with two in five authors saying critical analysis 
was their primary objective. About one author in four cited formulating 
general statements and identifying central issues as the primary goal and • 
about one in ten cited resolving conflicts or bridging gaps between 
theories or Ideas as their paraimKint interest. / 

The perspective category was dominated by authors who said they tool; a 
dispassionate view of the literature {6\%) and the organization of most 
reviews was conceptual (76%). 

About two of every three reviewers said they based their conclusions 
on all of the relevant material and about half of these said all the 
material was cited in their paper. About one in five revian^rs said they 
used a representative coverage strategy and one in ten a central or pivotal 
coverage strategy. 

About a third of the papers were directed toward specialized scholars, 
a third toward general scholars, or a third toward practitioners. Policy 
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makers and the general public were rarely the audience of reviews 
catalogued by the two abstracting services. 

Because of the interest surrounding meta-analysis, I thought It would ^ 
be informative to determine the percent of reviews that might be considered 
legitimate candidates for quantitative synthesis. About ^ne reviewer in 
six (17.6%) claimed their papers primarily focused on researchNiu^cpmes and 
had as a goal the formulation of general statements from multiple Specific 
instances. This might be considered a broad definitibn of a meta-analysis 
candidate. If we also Include in the definition that the author intended \ 
•to be neutral in perspective and to base conclusions on exhaustive 
literature coverage then the nimiber of reviews ripe for metalanalys1$ was 
one in eight (13%). This. finding can be Interpreted in two wjays. F^rst, 
advocates of meta-analysis can claim that their techniques \are applicable 
to the largest Intersection of review foci and goals. At the same time, 
however, this type of review represents only a small portion i>f all 
literature reviews. The ^rvey, therefore, indicates that other aspects of 
literature reviewing should not be neglected because of inordinate 
attention paid to Issues surrounding quantitative synthesis. 

To discover any relations between the different characteristics of 
reviews, I performed a correlational and factor analysis of the reviewers' 

♦ 

responses. *l will only describe some of the results in the most general 
terms. All the correlations I cite fell between r = .25. and r - .5 and 
reached at least the .01 level of significance. 

First, reviewers tended to view the characteristics of perspective and 
organization as containing mutually exclusive categories. This was 
evidenced by negative intra-characteristic correlations, by limited use of 
secondary rankings, and by unsolicited coiments from respondents. With 
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regard to qoverage, an exhaustive strategy was seen as exclnsive of all 
others. However, use of an exhaustive st^^tegy with both representative 
and central citations appeared frequently (r = .28), as xlld the two * 
selective strategies (r ~ .38). ^ 

The focus and goal categories revealed some positive 

. ■ 

intra-c^aracteristic relations. Reviews that focused on research methods 

I- 

also tended to foctis on research outcoines (r = .41) or .theories (r = .41). 
The goal of resolving, conflicts in the literature frequently^ppeared in 
conjunction with either formulating general statspnents (r * .41) or 
bridging theoretical gaps (r = .41). Critical analysis frequently appeared 
with identifying central issues (r « .28). 

With regard to inter-characteristic relations, a focus on research 
outcomes was associated with the goals of fonmjiating generalities 
(r = .49) an^ resolving conflict {r = .36) while writijfjg for an audl^qe of 
either specialfzed (r = .29) or general scholars (r = .36). Focusing on 
methods was associated with critical analysis (r .29) and identifying 
central issues (r = .33) as goals, a methodological organization (r » .29), 
and writing for general scholars (r « .35). A theoretical focus was 
associated with selectively covering works that <were representative of the 
literature (r = .36) and writing for general scholars (r = .25). 

A goal of formulating generalizations was .associated with/ exhaustive 
coverage but selective citation (representative citation, r ='.27; central 
citation, r = .25) and with an audience of scholars (specialized, r = .28; 
general, r = .30). Bridging thepretical gaps as a goal co varied with 
selective citation (representative citation, r == .30; central citation, 
r = 2.6), a historical organization (r -,.34) and either a general scholar 
(r * .32) or policyHRaker (r = .34) audience. 
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While these associations appear Intuitively appealing they should not- 
lead to a conclusion that a small nimber of similarly-structured prototypes 
under ly most reviews. The ^a):tor analysis I performed revealed a first 
principle component explaining only five.percent of the variance and ten 
factors with eigenvalues greater than one. Beyond the integrative research 
review, which I estimate accounts for no more than 20% of the review 
population* no frequently occurring pattern of multiple review 
characteristics was discernible. Both the descrlptl^ and relational data 
^ reveal a body of scholarship, called literature review, that Is diverse and 
held together only by the broadest' tenets of secondary analysis and 
critical synthesis that form- the general definition. - 

using the Taxonoiny to Help Judge the Quality of Reviews 

Perhaps the most perplexing question stemning fnm the increased 
dependence on literature reviews as a source of information concerns how to 
distinguish good reviews from bad ones.- Ue'have seen that diverse types of 
reviews exist and there is no reason to believe one type is intrifisical ly 
more valuable or valid than another. General discussions of review 
quality, therefore, will employ criteria of a highly abstract nature. • 
leaving much to the jud^nt of the' individual assessor'. 

Strike and Posner (1983) suggested that the question of synthesis 
quality has two part;. The first part involves the intellectual quality 
and soundness of the synthesis and the second involves its utility. With 
regard to Intellectual quality. Strike and Posner' of fered three criteria. 
First: ' 

A qual4ty synthesis will clarify and resolve, rather than 
obscure, inconsistencies or tensions b^ween material 
synthesized, (p 356-357) 
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Second: 

'a quality synthesis will result in a progressive problem shift 
... exh1bit[1ng] such features as increased explanatory and 

7 

predictive power and expanded einpirical cootent, increased 
theoretical ability to explain ideas synthesized, expanded scope 
of application, and an Increased capacity to Identify and pursue 
unsolved problems, (p. 357) 
And finally: • • * 

A successful synthesis will satisfy the fomSil crl.teria for good 
theories. Such standards as consistency, parsimony,. elegance, 
. and fruitful ness characterize a good synthesis, (p. 357) 
Strike and Posner's (1983) criteria are Indisputable elements of good 
syntheses. The difficulty In applying them, however, goes beyond the fact 
that they involve a great deal of subjective judgment. Especially for the 
(Triterla of resolving conflict and creating progressive problem shifts, the 
ability to assess whether a reviajr has performed these functions may take 
years to develop, since they are dependent on the Impact the synthesis .has 
on a field, rather than solely on the intrinsic qualities of the synthesis 
itself. The third criterion. Involving cohs1sten<^y, parsimony, and 
elegance, strikes closer to th^ kind of criteria individual readers will 
apply when they first encounter a review. 

There is some empirical evidence- that readers, of reviews do, in fact, 
intuitively employ something like Strike and Posner's (1983) third 
criteria. In a recent study of quality criteria for research reviews 
(Cooper, in press), i asked fourteen graduate students in education and 
psychology to read six reviews on the effect of desegregation on black 
student's achievement. The six reviews had been written simultaneously as 
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part of a panel sponsored by NlE's Desegregation Studies Team. First, 
readers were asked to make jiid9inents concerning five aspects of each 
review: the clarjty of pco|)lein definition, the exhaustiveness of the 
covered Vesearch, the validity of the reviewers' evaluations of the covered 
studies, the quality of the synthesis, and the clarity of writing. Factor 
atYalyses revealed that a single quality component accounted for between 58X 
and 83t of trte variance Jn the f^ve dimensions when the analysis was 
conducted separately for each review. A standardized composite of the five 
Judgments correlated between r ° .84 and .96 with the readers' overall 

judgnient of a review's quality. It might be concluded, then, that reviews 

. . « • ■ ■ ~ ■• . 

that were seen a$ strong on one qtjia1itj(. dimension also tended to be seen as 
strong orv others. 

A second analysis related the readers' quality judgments to their 
perceptions of the^ rev,iewer's. position .concerning the effects of 
desegregation. Somewhat surprisingly, quality judgments were not related 
to the reviewers' or readers' position on the desegregatton issue. 
Instead, the quality of a review was positively correlated 4^1 th the 
readers' confidence in. how clearly they could interpret where the reviewer 
stood on the issue. ' More Interpretability meant higher quality ratings. 

The preeminence of presentatibn factors' oh quality Judgments was 

■■•.■■% r ■ 

further substantiated %y open-ended evaluative* comments supplied by the 
readers. In these comments, the readers most often mentioned that a paper 
was either well or poorly organized. Second most frequerftly mentioned was- 
writing style, in particular the author's ability or inability to keep the 
interes(,,of the reader. * Jhird was how well or poorly focused the paper was 
on the desegregation issue and fourth was how well or poorly the reviewers 
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usfed citations to substantiate their claims. All of these criteria appear 
to be congruent with Strike and Posner's third notion. ' 

In sum, then, the general criteria for good literature reviewing range 
from tJje lofty pursuits of resolving conflict and stimulating progressive 
problem' shifts to the rather ^own-to-earth concern of presenting material 
effectively enough to get one's point across. Delineating such abstract 
criteria, however, may not clarify the problem of how to judge quality as 
much as make potential reviewers wary about pursuing such a complex task. 

The taxonomy, I think, can help niake evaluating reviews more 
manageable. Strllce and Posner's (1983) second part to assessing quali1;y 

< 

Involved utility. In their words, "... useful syntheses will be 
syntheses which answer the question asked" (p 357-358). This can be 
translated Into two questions Involving the six characteristics of reviews. 
First, do the foci, goals, perspective, coverage, organization and audience 
of the review form a logical whole? Second, does the review attend to the 
foci, Rieet the goals, and employ the expository design the reviewer set for 
It? I will examine each question in turn. , 

With regard to the logic of a review^ we cao ask whether an author has 
chosen a set of characteristics that are Internally consistent. For 
instance, reviewers who establish' the goal of integrating research to form 
general statements are being inconsistent If they couple this objectfve 
with a selective coverage of the literature. Likewise, exhaustive citation 
of a literature wuld be counter-productive for a review with the goal of 
identifying central Issues, or for one written for practitioners or policy 
makers. Obviously, assessing the congruence of matchlngs could go on. 
While I would have liked to provide a complete list of what characteristics^ 
do artd don't fit together, I don't believe such a cataloging Is possible. 
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Some matchings may make sense for certain topic areas but not for others. 
The point is that having a common, structured scheme for discussing the 
characteristics of reviews allows assessment and debate of matchings' that 
do occur. 

The taxonoB^ also allows readers to more comprehensively judge whether 
a review did what it set out to do. An author who claims to have written a 
review of practices, meant to identify central issues, from a dispassionate 
perspective, and covering all the relevant literature, provides readers 
with several self-imposed standards. The key to the use of the taxonoiny in 
this fashion, of course, lies in the willingness of reviewers to state 
explicitly what they are up to. As we have seen, for certain 
characteristics such clarity has not prevailed in the past. I would 
suggest that more important than the creation of any single, uniform schUltR 
for describing reviews is that authors of reviews thoroughly describe the 
intent and nature of their work, in whatever terms make them comfortable. 

The Importance of authors describing their inteatlons and practices 

4 

recently became personally salient to me when I was asked to be a member of 
this year's AEM Research Review Award Committee. Being a committee meRd)er 
gave me the opportunity to examine my own process of review evaluation. As 
I read througb the numerous candidates, I found I. could not even begin the 
task until I thought I understood why and how a review was being carried 
out. Next, I mused over whether the -foci, goals, and procedures of the 
review contained a credible internal logic. These two judgments were 
minimum criteria. If a review did not meet them I could not give it 

further consideration. The next st^ Involved a Judgment of the complexity 

^- 

or difficulty of the task set by the reviewer. Thus, goals became an 
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important, discriminating criterion. In general, I found linguistic 
bridge-building and conflict resolution were more lofty pursuits than 
generalization, criticism, or Issue identification, though reviews that 
attempted to Integrate large bodies of literature relating to broad 
concepts Were also deemed very worthwhile. The final criteria concerned 
how well the authors accomplished the goals they set for themselves. Thus, 
the evaluation was much like iralghting the performance of a dive or 
gymnastic routine by its degree of difficulty. The conjunction of the last 
two judgments, I think, was meant to predict urtilch reviews were most likely 
to achieve Strike and PQsner's (1983) criterion of precipitating a 
progressive problem shift. 

If vauthors make thein aims, and procedures clear, the process of 
judging quality not only becomes* more feasible, but It can be more 
objective, as well. Presently, this Increased objectivity is evident in 
the area of integrative research reviews. For example, earlier I defined a 
meta-analysis as art integrative review of research outcomes, seeking 
generalittes, synthesizing the entire relevant literature In a 
"dispassionate fashion. A review with such characteristics can be held up 
to a fajriy explicit and objective set of standards. Establishing these 
criteria h^s occupied this author's attention for several years (see 
Cooper,. 1984|). Some likely candidates appear In Figure 2. Questions that 
can be asked about Integrative research reviews Include: (a) do the 
operations appearing In the literature fit the review's abstract 
definitions?; (b) Is enough attention paid to the methodological details of 
stud1es?4> (c) was the literature search thorough?; (d) were studies 
evaluated using explicit and consistent rules?; and (e) were valid 
procedures used to conH>ine the results of separate studies? Because the 
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process of evaluating integrative research reviews parallels that of 
evaluating primary research, the establishment of quality criteria for 
these reviews is somewhat easier than for other types of syntheses. It is 
important to recognize that, beyond the general criteria discussed earlier, 
no set of specific rules will apply to all types of reviews. Each cluster 
of review characteristics will require a set of distinct standards. 



Place Figure 3 about, here 

— r 

Other Uses for the Taxonomy 
. Before coi^luding, there are two other Important use for the taxonos^. 
First, many editors of books and journals, directors of information 
clearinghouses, arid funding agencies solicit reviews from particular 
authors or solicit review proposals. The taxonomy can be used by editors 
or agencies to communicate to potential authors what they are after. This 
can be especially helpful if a review is meant to fulfill a pdrtfcular need 
or if multiple reviews on the sane topic are being solicited and each 
' review is meant to take a different approach. I will shortly be developing 
for the ERIC clearinghouses just such an application for the taxonony, 
along with a means for evaluating whether using the taxonomy t6 direct, 
authors' efforts enhances readers' perceptions of the utility of the 
Information Analysis Products sponsored by the Clearinghouses. 

Finally, the taxonon^ can be used as a framework for graduate 
education courses in literature reviewing. Students in education and 
psychology literally can take five or six statistics and methods courses 
without ever directly addressing the problems and procedures of literature 
review. This situation ^ slowly changing. Hopefully, the taxonoR^ will 
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facilitate the development of such coursed by suggesting a course structure 
and relevant issues that courses need to address. 

Conclusion 

I mentioned earlier that this paper was a wrking document meant to 
s-tiimilate discussion. My major aims have been to interest others In a 
topic that has gained increasing salience in education and psychology and 
to fraipe some of the questions the topic has engendered^ Hy own 
involvement with literature reviews began with the circumscribed task of 

* 

how to best combine the statistical results of independent empirical 
studies. Shprtly thereafter, I came to believe the literature search was 
at least as, if not more, crucial to the outcome of^ research reviews than^ 
the synttv^sls process itself. Finally, I saw that research synthesis was 
only oiie species of a broad genus. The genus, however, existed without 
organizing principles. As both consumers and producers of reviews, , 
education and psychology researchers could only benefit from others 
improving on ny "fleldglasses" and "guidebook". 
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Footnotes 

^ For Psych Info , every fourth reviet^er on the computer printout was 
not contacted. For ERIC , every fourth and fifth reviewer was passed over. 
If an address for a first author could not be found, we returned to the 
beginning of the list and repeated the procedure. Much more diff^iculty was 
encountered in locajting ERIC authors— the entire listing was exhausted in 
• obtaining the 65 authors sampled, this was because ERIC contains more 
documents by doctoral cahdidates and by authors not affiliated with 
4jn i vers i ties. Our primary sources of addresses were: (a) the publication 
itself; (b) professional organization directories (i.e., APA and AERA); and 
(c) directories of Anericah University faculty members. 
2 

Correlations and factor analyses were perfonned on data converted to 
reflect whether or not a category was mentioned by a reviewer, regardless 
of its ranking. Thus, if a category received any rank it was given a value 
of 1, if it was omitted it was assigned a value of 0. A second set of 
analyses that retained the ranking distinctions but treated then as 
interval rather than ordinal data produced results similar to those 
described above. 
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f IGURE 1 
(Figure Caption) 

Figure 1 , Percentage of doc4JBents described as literature reviews in the 
Psychological Abstracts and ERIC Databases. 



Notes : ^ Psychological Abstracts data is based on the percentage of total 

docqments assigned the descriptors "Literature Review" or "Review of 
the Literature." 

^ 'eriC data is based on the percentage of total docuaients assigned the 
descriptors "Literature Review" or "Research Reviews." Totals 
include both RIE(ED) and CIJE(EJ} documents. In 1980. ERIC 
Instituted a document type designation for literature reviews (070). 
The use of the descriptor "Literature Review" was deemphasized and 
the <fescriptor "Research Review" was eliminated. 
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* TABLE 1 
A TaxonoB^y of Literature Reviews 



Characteristic 



Categories 



Focus 



Goal 



Perspective 
Coverage 



Organization 



Audience 



Research Outcomes 
Research Methods 
Theories 

Practices or Applications 

Integration 

a] Generalization 

b) Conflict Resolution 

> c) Linguistic Bridge-build'ing 
Criticism 

Identification of Central Issues 

Neutral Representation 
Espousal of Position 

Exhaustive 

Exhaustive with Selective Citation 

Representative 

Central or Pivotal 

Historical 
Conceptual 
Methodological 

Specialized Scholars 
General Scholars 
Practitioners or Policy Makers 
General Public 
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TABLE 2 

■J 

A Categorization of Reviews Winning AERA's Research Review Award (1978-1984) 



Year of Award and First Author 



Characteristic 



1984 
««bb 



1983 
St1p«k 



1^82 
Thomas 



1981 
Boruch 



.1980 
Paul son 



1979 
Vltkin 



1978 
Shaval son 



Focus 



Outcoows, 
Nothods 



OutCOMS (3) 
Theorlas (2) 



Outcoows (3) Practices (3) Outcoows (2) OutcooMs (3) Nothods (3) 

Practfcas (2) Methods (2) Practices {%) Theories (1) 



Coal 



integratlm (3) Integration (3) Central Issues (2) Central Issues (3) Central Issues (3) Central issues (2) ,Crft1c1s« (2) 

Integration (2) Central Issues (2) 



Perspective 



Cmtral issues 13) Central Issues C^) Critlclsa (2) Integratloi) (3> integration (2) 
^Itautra) (3) Nratral (3) EsptMital (3) Iteutrel (3) Neutral (3) 



Espousal (2) Espousal (2) 



Coverage 



Exhaysttve (3) ExhMJstfve (3) Selactlve (3) Salectlva (2) Salactlve (2) Reprasent (2) Salectlva (2) 

Rapresant (3) Represent (2) Represent (2) 



Organfzatfon ^ Conceptual (3) 



Audience Specialized (3), 



CoMeptual (3) Conceptual (3) Conceptual (2) ^Conceptual (3) Conceptual (2) (^rations (2) 

Operations (2) Historical (2) Historical (2) Conceptual (2) 

Specialized (3) Practitioner (2) Policy Makers (3) Specialized (2) Specialized (2) Specialized (3) 

Practitioner (2) Practitioner (2) 
denertil (2) 



i 
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TABLE 3 ' 

Rellabili,tle8 and Agreement Rates for Two Users 6t the Taxonomy 



First Code First Plus Second Code 

Cohen's K % Agreeaent Cohen's K Z Aggr extent 



Focus 


.48 


68Z 


.55 


i 


65Z 


Goal 


.20 


48Z 






86Z 

ii 


Perspective 


.53 


78Z 


no 


second 


codes 


Coverage 


.32 


49Z 


few 


second 


codes 


Organization 


.23 


i>4Z 


.45 






Audience 


.33 


84Z 


.60 




73Z 
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FIGURE 2 
(Figure Caption) 

Figure 2 . The Integrative Review Conceptualized as a Research Project 



Reprinted with permission fro» Cooper, H. Scientific guidelines for 
conducting integrative research reviews. Review of Educational Research , 
1982, 52, 291^302. Copyright 1982 by the American Educational Research 
Association. 
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TABLE 4 

Percentages of Authors Using Various Categories to Describe Their Reviews 



Characteristic . Categories 

Percentage of Authors Using as Descriptor 







Priwary 


Secondary 


OAiltted 


Focus 


Research Outcomes 


56 


12 


25 




Kesearcn nethods . 


9 


. 17 


47 




Theories 


19 


23 


32 


• 


Practices or Applications . 

- 


22 


23 


26 


Goal . ^ 


Integration 










< a) Generalization 


23 


19 


35 




D) Conflict Resolution 


7 


11 


59 




c) Linguistic Bridge- bull ding 


8 


6 


68 




Crltlctsw 


42 


16 


24 




Identification of Central Issues 


24 


21 


34 


Perspective 


Neutral Representation 


81 




14 


• 


Espousal of Position 

- ■ 


18 


18 ^ 


65 


Coverage 


Exhaustive 


37 


2 


52 


- 


Exhaustive with Representative Citation 


21 


4 


70 




Exhaustive with Central Citation 


14 


7 


75 




Representative 


19 


6 


70 




Central or Pivotal 


7 


6 


78 


Organization 


Historical 


7 


9 


77 




Conceptual 


76 


6 


17 




Methodological 


15 


13 


70 


Audience 


Specialized Scholars 


39 


26 


28 




General Scholars 


34 


23 






Practitioners 


31 


18 






Policy Makers 


4 


5 






General Public 


2 
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